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Abstract 

The aim of this paper is to prove the achievability of several coding problems by using sparse matrices (the 
maximum column weight grows logarithmically in the block length) and maximal-likelihood (ML) coding. These 
problems are the Slepian-Wolf problem, the Gel'fand-Pinsker problem, the Wyner-Ziv problem, and the One- 
helps-one problem (source coding with partial side information at the decoder). To this end, the notion of a hash 
property for an ensemble of functions is introduced and it is proved that an ensemble of q-ary sparse matrices 
satisfies the hash property. Based on this property, it is proved that the rate of codes using sparse matrices and 
maximal-likelihood (ML) coding can achieve the optimal rate. 

Index Terms 

Shannon theory, hash functions, linear codes, sparse matrix, maximum-likelihood eoncoding/decoding, the 
Slepian-Wolf problem, the Gel'fand-Pinsker problem, the Wyner-Ziv problem, the One-helps-one problem 

L Introduction 

The aim of this paper is to prove the achievabiUty of several coding problems by using sparse matrices (the 
maximum column weight grows logarithmically in the block length) and maximal-Ukelihood (ML) coding 
namely the Slepian-Wolf problem [39] (Fig. 1), the Gel'fand-Pinsker problem [13] (Fig. 2), the Wyner-Ziv 
problem [47] (Fig. 3), and the One-helps-one problem (source coding with partial side information at the 
decoder) [44] [46] (Fig. 4). To prove these theorems, we first introduce the notion of a hash property for an 
ensemble of functions, where functions are not assumed to be linear. This notion is a sufficient condition for 
the achievability of coding theorems. Next, we prove that an ensemble of q-aiy sparse matrices, which is an 
extension of [21], satisfies the hash property. Finally, based on the hash property, we prove that the rate of 
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'This operation is usually called an ML decoding. We use the word 'coding' because this operation is also used in the construction of 
an encoder. 
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codes can achieve the optimal rate. This implies that the rate of codes using sparse matrices and ML coding 
can achieve the optimal rate. It should be noted here that there is a practical approximation method of ML 
coding by using sparse matrices and the linear programing technique introduced by [11]. 
The contributions of this paper are summarized in the following. 

• The notion of a hash property is introduced. It is an extension of the notion of a universal class of hash 
functions introduced in [7]. The single source coding problem is studied in [22, Section 14.2][17] by 
using the hash function. We prove that an ensemble of g-ary sparse matrices has a hash property, while 
a weak version of hash property is proved in [26] [2] [10] [3 3] [34] implicitly. It should be noted that our 
definition of hash property is also an extension of the definition of random bin coding introduced in [3], 
where the set of all sequences is partitioned at random. On the other hand, the random codebook (a set of 
codewords/representations) generation is introduced for the proof of the original channel coding theorem 
[37] and lossy source coding theorem [38]. Here it is proved that random bin coding and partitioning 
determined by randomly generated matrix can be applied to the original channel coding theorem and lossy 
source coding theorem. 

• The proof of the achievability of the Slepian-Wolf problem is demonstrated based on the hash property. 
It is the extension of [22, Section 14.2][17] and provides a new proof of [3][5][33]. By applying the 
theorem to the coding theorem of channel with additive (symmetric) noise, it also provides a new proof 
of [26][2][10][33]. 

• The optimality of a code is proved by using sparse matrices and ML coding for the Gel'fand-Pinsker 
problem. We prove the g-ary and asymmetric version of the theorem, while a binary and symmetric 
version is studied in |24|. It should be noted here that the column/row weight of matrices used in [24] is 
constant with respect to the block length, while it grows logarithmically in our construction. The detailed 
difference from [24] is stated in Section V-B. As a corollary, we have the optimality of codes using sparse 
matrices for the coding problem of an arbitrary (g-ary and asymmetric) channel, while a symmetric channel 
is assumed in many of the channel coding theorems by using sparse matrices. The construction is based on 
the coset code presented in [33][29], which is different from that presented in [12][2]. When our theorem 
is applied to the ensemble of sparse matrices, our proof is simpler than that in [29]. 

• The optimality of a code is proved by using sparse matrices and ML coding for the Wyner-Ziv problem. 
We prove the g-ary and biased source, and the non-additive side information version of the theorem, while 
a binary and unbiased source and additive side information are assumed in [24]. As a corollary, we have 





Rx > H{X\Y) 
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Fig. 1. Slepian-Wolf Problem 
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Fig. 2. Gel'fand-Pmsker Problem 
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Fig. 3. Wyner-Ziv Problem 



the optimality of codes using sparse matrices for the lossy coding problem of an arbitrary (g-ary and 
biased) source and a distortion measure. In [25J[36J[23J[27J[14J, a lossy code is proposed by using sparse 
matrices called low density generator matrices (LDGM) by assuming an unbiased source and Hamming 
distortion. The column/row weight of matrices used in [24] is constant with respect to the block length, 
while it grows logarithmically in our construction. The lower bounds on the rate-distortion function is 
discussed in |8|| 19]. It should be noted that the construction of the codes is different from those presented 
in [25] [36] [23] [24] [27][14|. The detailed difference is stated in Section V-C. Our construction is based 
on the code presented in [28] [30] and similar to the code presented in [42][43||49|. When our theorem is 
applied to the ensemble of sparse matrices, our proof is simpler than that in [28]. 
• The achievability of the One-helps-one problem is proved by using sparse matrices and ML coding. 

II. Definitions and Notations 
Throughout this paper, we use the following definitions and notations. 

Column vectors and sequences are denoted in boldface. Let Au denote a value taken by a function A : — > 
U &i u & U"' where is a domain of the function. It should be noted that A may be nonUnear. When A is 
a linear function expressed by an Z x n matrix, we assume that U = GF(g) is a finite field and the range of 
functions is defined by U = UK It should be noted that this assumption is not essential for general (nonUnear) 
functions because discussion is not changed if Zlog|ZY| is replaced by log|W|. For a set A of functions, let 
Im^ be defined as 

Im^= IJ {Au : uGU"}. 

The cardinality of a set U is denoted by \U\, W denotes the compliment of U, and U\V = U (IV denotes 
the set difference. We define sets Ca{c) and Cab{c, b) as 

Ca{c) = {u : Au = c} 

Cab{c, b) = {u : Au = c, Bu = b}. 

In the context of Unear codes, Ca(c) is called a coset detemnined by c. 
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Fig. 4. One-helps-one Problem 



Let p and p' be probability distributions and let q and q' be conditional probability distributions. Then entropy 
H{p), conditional entropy H{q\p), divergence D{p\\p'), and conditional divergence D{q\\q'\p) are defined as 

H{p) = J2piu) log^ 
H{q\p) = q{u\v)p{v) log 

u,v ^ ' ' 

Dip\\p')^Ypiu)log^ 
D{q II q'\p) ^ E^'(^)E«("l^)log5S' 

V U I \ I / 

where we assume the base 2 of the logarithm when the subscript of log is omitted. 

Let nuv be the joint probabihty distribution of random variables U and V. Let fjbu and /xy be the respective 
marginal distributions and Hu\v be the conditional probabihty distribution. Then the entropy H{U), the 
conditional entropy H{U\V), and the mutual information I{U ; V) of random variables are defined as 

H{U) = H{fiu) 

H{U\V)^H{iiu\vW) 

I{U;V) = H{U)-H{U\V). 

A set of typical sequences Tu^^ and a set of conditionally typical sequences 7[/|y^^(t;) are defined as 

Tu^^ = {u : D{iyu\\nu) < 7} 
'^u\v,'r{v) = {u : DiPulvWlJ-uivWv) < 7} , 

respectively, where Uu and v^iv defined as 

f . \{l<i<n:Ui = u}\ 
n 

We define x(') as 

X{a = b) = 
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Finally, for 7, 7' > 0, we define 



_ |Z^|log[n+l] 
n 



Cw(7)=7-\/2^1og^ (2) 
Qu\v{i\l) =i- v^log + log (3) 



. X A^, |W|log[n + ll 
^(7)--^log^ + ^^ i 



(4) 



r^,v(7'l7) - -^/^log^ + ^loglZYl + m^^^^-^^y (5) 

|W||V| n 

It should be noted here that the product set Z/^ x V is denoted by UV when it appears in the subscript of these 
functions. 

III. (a, /3)-HASH Property 

In this section, we introduce the notion of the {a, /3)-hash property which is a sufficient condition for coding 
theorems, where the linearity of functions is not assumed. The (Q;,/3)-hash property of an ensemble of linear 
(sparse) matrices will be discussed in Section IV. In Section V, we provide coding theorems for the Slepian-Wolf 
problem, the Gel'fand-Pinsker problem, the Wyner-Ziv problem, and the One-helps-one problem. 

Before stating the formal definition, we explain the random coding arguments and two implications which 
introduce the intuition of the hash property. 

A. Two types of random coding 

We review the random coding argument introduced by [37]. Most of coding theorems are proved by using 
the combination of the following two types of random coding. 

Random codebook generation: In the proof of the original channel coding theorem [37] and lossy source 
coding theorem [38], a codebook (a set of codewords/representations) is randomly generated and shared by the 
encoder and the decoder It should be noted that the randomly generated codebook represents the list of typical 
sequences. In the encoding step of channel coding, a message is mapped to a member of randomly generated 
codewords as a channel input. In the decoding step, we use the maximum-likelihood decoder, which guess the 
most probable channel input from the channel output. In the encoding step of lossy source coding, we find a 
member of randomly generated representations to satisfy the fidelity criterion compared to a message, and then 
we let the index of this member as a codeword. In the decoding step, the codeword (index) is mapped to the 
reproduction. It should be noted that the encoder and the decoder have to share the large size (exponentially 
in the block length) of table which indicates the correspondence between a index and a member of randomly 
generated codewords/representations. The time complexity of the encoding and decoding step of channel coding 
and the encoding step of lossy source coding is exponentially in the block length. They are obstacles for the 
implementation. 

Random partitioning (random bin coding): In the proof of Slepian-Wolf problem [3], the set of all sequences 
is partitioned at random and shared by the encoder and the decoder. In the encoding step, a pair of messages 
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Fig. 5. Properties connecting the number of bins and items (black dots, messages), (a) Collision-resistant property: every bin contains 
at most one item, (b) Saturating property: every bin contains at least one item, (c) Pigeonhole principle: there is at least one bin which 
contains two or more items. 

are mapped independently to the index of bin which contains the message. In the decoding step, we use 
the maximum-likelihood decoder, which guess the most probable pair of messages. Random partitioning by 
cosets determined by randomly generated matrix can be considered as a kind of random bin coding, where 
the syndrome corresponds the index of bin. This approach is introduced in [9] for the coding of symmetric 
channel and apphed to the ensemble of sparse matrices in [26] [2] [10]. This argument is also applied to the 
coding theorem for Slepian-Wolf problem in [45] [5] [33]. It should be noted that the time complexity of the 
decoding step is exponentially in the block length, but there are practical approximation methods by using 
sparse matrices and the techniques introduced by [1][18][11]. By using the randomly generated matrix, the size 
of tables shared by the encoder and the decoder has at most square order with respect to the block length. 

One of the aim of introducing hash property is to replace the random codebook generation by the random 
partitioning. In other words, it is a unification of these two random coding arguments. It is expected that the 
space and time complexity can be reduced compared to the random codebook generation. 

B. Two implications of hash property 

We introduce the following two impUcations of hash property, which connect the number of bins and messages 
(items) and is essential for the coding by using the random partitioning. In Section III-D, these two property 
are derived from the hash property by adjusting the number of bins taking account of the number of sequences. 
Collision-resistant property: The good code assigns a message to a codeword which is different from the 
codewords of other messages, where the loss (error probabiUty) is as small as possible. The colhsion-resistant 
property is the nature of the hash property. Figure 5 (a) represents the ideal situation of this property, where 
the black dots represent messages we want to distinguish. When the number of bins is greater than the 
number of black dots, we can find a good function that allocates the black dots to the different bins. It 
is because the hash property tends to avoid the collision. It should be noted that it is enough for coding 
problems to satisfy this property for 'almost all (close to probabiUty one)' black dots by letting the ratio 
[the number of black dots] /[the number of bins] close to zero. This property is used for the estimation of 
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decoding error probability of lossless source coding by using maximum-likelihood decoder. In this situation, 
the black dots correspond to the typical sequences. 

Saturating property: To replace the random codebook generation by the random partitioning, we prepare 
the method finding a typical sequence for each bin. The saturating property is the another nature of the hash 
property. Figure 5 (b) represents the ideal situation of this property. When the number of bins is smaller than 
the number of black dots, we can find a good function so that every bins has at least one black dot. It is 
because the hash property tends to avoid the collision. It should be noted that this property is different from the 
pigeonhole principle: there is at least one bin which includes two or more black dots. Figure 5 (c) represents 
an unusual situation, which does not contradict by the pigeonhole principle while the hash property tends to 
avoid this situation. It should be noted that it is enough for coding problems to satisfy this property for 'almost 
all (close to probability one)' bins by letting the ratio [the number of bins] /[the number of black dots] close to 
zero. To find a typical sequence from each bin, we use the maximum-likelihood/minimum-divergence coding 
introduced in Section III-D. In this situation, the black dots correspond to the typical sequences. 

C. Formal definition of (a, l3)-hash property 
In this section, we introduce the formal definition of the hash property. 

In the proof of the fixed-rate source coding theorem given in L3J[5J[17J, it is proved implicitly that there is 
a probability distribution pA on a set of functions A : ZY" — > ZY' such that 

PA {{A ■.3u'eg\ {u}, Au' = Au}) < M (6) 

for any u gW^, where 

g = {u' : n{u') > ii{u), u i- u'} (7) 

and \i is the probability distribution of a source or the probability distribution of the additive noise of a channel. 
In the proof of coding theorems for spare matrices given in [2][10][26][33][34], it is proved impUcitly that there 
are a probability distribution on a set of Z xn spare matrices and sequences a = {a(n)}5^i and /3 = 
satisfying 

lim i^^ = 

n— >oo n 

lim /3(n) = 

such that 

PA {{A : 3u' e a \ {n}, Av! = Au}) < + /?(n) (8) 

for any u E W", where a{n) measures how the ensemble of / x n sparse matrices differs from the ensemble 
of all / X n matrices and fi{n) measures the probability that the code determined by an / x n sparse matrix has 
low- weight codewords. It should be noted that the collision-resistant property can be derived from (6) and (8). 
It is shown in Section III-D. 

The aim of this paper is not only unifying the above results, but also providing several coding theorems 
under the general settings such as an asymmetric channel for channel coding and an unbiased source for lossy 
source coding. To this end, we define an (a,/3)-hash property in the following. 
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Definition 1: Let ^ be a set of functions A : U"- and we assume that 

lim ^^^S = 0. (HI) 

n^oo n 

For a probability distribution ^ pA on A, we call a pair {A,pa) an ensemble Then, an ensemble {A^pa) has 
an (ptA,(^A)'hash property if there are two sequences = {a^(n)}5^i and = {/^^(n)}^^ ^^ch that 

lim aA(n) = 1 (H2) 
lim (3A{n) = 0, (H3) 

n—*oo 

and 

PA {{A : Au = Au']) < |T n T'| + '^"^^^^^"^ + min{|T|, |T'|}/3^(n) (H4) 

u'It' 

for any T, T' c ZY". Throughout this paper, we omit dependence on n of a a and when n is fixed. 

It should be noted that we have (8) from (H4) by letting T = {u} and T' = Q, and (6) is the case when 
a^(n) = 1 and (iA{n) = 0. In the right hand side of the inequaUty (H4), the first term corresponds to the 
sum of pa{{A : Au = Au}) = 1 over all u e T n T', the second term bounds the sum of the probabiUty 
Pa{{A : Au = Au'}) which are approximately l/|Inx4| for u ^ u', and the third term bounds the sum of 
the probability pa{{A : Au = Au'}) far greater than l/|Im^| for u ^ u'. This intuition is explained in 
Section IV for the ensemble of matrices. 

In the following, we present two examples of ensembles that have a hash property. 
Example 1: Our terminology 'hash' is derived from a universal class of hash functions introduced in [7]. We 
call a set A of functions A : Ua a universal class of hash functions if 

\{A:Au = Au'}\<^ 

for any u ^ it'. For example, the set of all functions on and the set of all linear functions A : 
are classes of universal hash functions (see [7]). Furthermore, for = GF(2"), the set 

Au = [the first 1^ bits of au] 



A 



A: 

a e GF(2") 



is a universal class of hash functions, where au is a multiplication of two elements a,u G GP(2"). It should 
be noted that every example above satisfies Im^ = Uj,. When A is a class of universal hash functions and pa 

^ It should be noted that pa does not depend on a particular function A. Strictly speaking, the subscript A of p represents the random 
variable of a function. We use this ambiguous notation when A aperars in the subscript of p because random variables are always denoted 
in Roman letter. 

'in the standard definition, an ensemble is defined as a set of functions and a uniform distribution is assumed for this set. It should be 
noted that an ensemble is defined as the probabiUty distribution on a set of functions in this paper. 

■* It should be noted that oca and I^a do not depend on a particular function A but may depend on an ensemble {A,pa)- Strictly 
speaking, the subscript A represents the random variable. 
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is the uniform distribution on A, we have 

J2 PA {{A : An = An'}) <\Tn T'| + 

This implies that {A,p) has a (1, 0)-hash property, where l(n) = 1 and 0(n) = for every n. 
Example 2: In this example, we consider a set of linear functions A : — > U^^ . It was discussed in the 
above example that the uniform distribution on the set of aU linear functions has a (1, 0)-hash property. The 
hash property of an ensemble of g-ary sparse matrices wiU be discussed in Section IV. The binary version of 
this ensemble is introduced in [21]. 

D. Basic lemmas of hash property 

In the following, basic lemmas of the (a,/3)-hash property are introduced. All lenmias are proved in 
Section VI-A. Let A (resp. B) be a set of functions A : ^ lij, (resp. B : ^ Kb)- We assume 
that {A,pa) (resp. {B,pb)) has an (aA,/3^)-hash (resp. (aB,/3B)-hash) property. We also assume that pc is 
the uniform distribution on 1mA, and random variables A, B, and C are mutually independent, that is, 

if c e Im^ 



Pc{c) 

PABc{A, B, c) = pa{A)pb{B)pc{c) 



|Im^| 

0, if ceUA\ l^A 



for any A, B, and c. 

First, we demonstrate that the coUision-resistant property and saturating property are derived from the (a, f3)- 
hash property. 

The first lemma introduce the collision-resistant property. 
Lemma 1: For any Q dU^ and u e ZY", 

PA {{A : [g \ {«}] n Ca{Au) + 0}) < + Ha. 

We prove the collision-resistant property from Lemma 1. Let /xy be the probability distribution on C/ C Z//". 
We have 

Ea \\Jiu ({m : \Q \ {«}] n Ca{.Au) ^ 0})] < Mu)pA {{A : [Q \ {u}] n Ca{Au) 7^ 0}) 

\g\aA 



ueg 



\lmA\ 



+ 13 J. 



By assuming that |5|/|Im^| vanishes as n — > 00, we have the fact that there is a function A such that 

Mc/ {{u : [g \ {u}] n Ca{Au) ^ 0}) < <5 

for any S > and sufficiently large n. Since the relation [g \ {u}] n Ca{Au) ^ corresponds to the event 
that there is u' E g, u' u such that u and u' are the members of the same bin (have the same codeword 
determined by A), we have the fact that the members of g are located in the different bins (the members of g 
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can be decoded correctly) with high probabiUty. In the proof of fixed-rate source coding, Q is defined by (7) 
for given probability distribution /if/ of a source U, where jiuiQ'^^) is close to zero. In the linear coding of a 
channel with additive noise, additive noise u can be specified by the syndrome Au obtained by operating the 
parity check matrix A to the channel output. It should be noted that, when Lemma I is appUed, it is sufficient 
to assume that 

lim i^i^=0 

n— *oo 77, 

instead of (H2) because it is usually assumed that |5|/|Im^| vanishes exponentially by letting n — > oo. It is 
implicitly proved in [26][2][10][33] that some ensembles of sparse linear matrices have this weak hash property. 
In fact, the condition (H2) is required for the saturating property. 

The second lemma introduce the saturating property. We use the foUoing lennma in the proof of the coding 
theorems of the Gel'fand-Pinsker problem, the Wyner-Ziv problem, and the One-helps-one problem. 

Lemma 2: Jf T ^ ^, then 

PAC {{{A, c) : T n Ca{c) = 0}) < - 1 + l^^^li^^ + ^l 
We prove the saturating property form Lemma 2. We have 

Ea \pc ({c : T n Ca(c) = 0})] = pac {{{A, c) : T n Ca{c) = 0}) 

By assuming that |Iin^|/|T| vanishes as n ^ c», we have the fact that there is a function A such that 

Pc{{c:TnCA{c)=^}) <6 

for any 6 > and sufficiently large n. Since the relation T n C^(c) = corresponds to the event that there is 
no u e T in the bin C^(c), we have the fact that we can find a member of T in the randomly selected bin 
with high probability. It should be noted that, when Lemma 2 is applied, it is sufficient to assume that 

lim l^iM^ = o 

n — *oo Jl 

instead of (H3) because it is usually assumed that |Imyl|/|T| vanishes exponentially by letting n — > oo. In 
fact, the condition (H3) is required for the collision-resistant property. 

Next, we prepare the lemmas used in the proof of coding theorems. The following lermnas come from 
Lemma 1. 
Lemma 3: If 5 C U"- and u ^ Q, then 

/ 

PAC 



gnCA{c)^9 

{A,c): 

u e Ca(c) 



^ \g\aA Pa 
|Im^|2 ^ |Im^| ■ 



< 



Lemma 4: Assume that ua,c S depends on A and c. Then 

PABC {{{A, B, c):[g\ {uA,c}] n Cab{c, Bua,c) 7^ 0}) < + /3b 

for any Q C W". 

Finally, we introduce the method for finding a typical sequence in a bin. The probability of an event where the 
function finds a conditionally typical sequence is evaluated by the following lemmas. They are the key lemmas 
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for the coding theorems for the Gel'fancl-Pinsker problem, the Wyner-Ziv problem, and the One-helps-one 
problem. These lemmas are proved by using Lemma 2. For £ > 0, let 



\og\U\ 

and assume that ^ is a set of functions A : Z^" — > U^^ and v G TV,^. 
Lemma 5: We define a maximum-likelihood (ML) coding function qa under constraint u G Ca{c) as 

gA{c\v) = arg max 
ueCAic) 

= arg max ij,uv{u,v) 
ueCAic) 

and assume that a set T{v) C ^|v,2£('y) satisfies 

• T(t>) is not empty, and 

• if u e T{v) and u' satisfies 

then u' e T(t)). 

In fact, we can construct such T{v) by taking up |T(i>)| elements from 'TmY2e{v) in the order of probability 
rank. If an ensemble {A,pa) of a set of functions A : U'^ has an (a^, /3^)-hash property, then 

VAC ({(A c) : gA{c\v) i T{v))) < - 1 + J ^^f§^ + 

for any u satisfying Tij\v,2ekv) 7^ 0- 
Lemma 6: We define a minimum-divergence (MD) coding function qa under constraint u e Ca(c) as 

?a(c|v) = arg mill D{vu\^\\iiu\v\i'v) 

= arg min D{uuv\\li'Uv) 

ueCAic) 

and assume that, for 7 > 0, a set T c Tij\y^ry{v) satisfies that if it e T and it' satisfies 

D{yu'\v\\l^u\v\^v) < D{v^\„\\^iu\v\yv) 

then u' e T. In fact, we can construct such T by picking up |T| elements from 7^|y,^(v) in the descending 
order of conditional divergence. Then 

PAC {{{A, c) : gA{c\v) ^ T{v)}) < - 1 + '^"''^|(^^|^ 

for any v satisfying Tu\^y^^{v) 7^ 0. 

In Section V, we construct codes by using the maximum-likelihood coding function. It should be noted that 
we can replace the maximum-likelihood coding function by the minimum-divergence coding function and prove 
theorems simpler than that presented in this paper. 
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"^(") = THuT ■ "^^^ 777— TV (9) 



IV. Hash Property for Ensemble of Matrices 
In this section, we discuss the hash property of an ensemble of (sparse) matrices. 

First, we introduce the average spectrum of an ensemble of matrices given in [2]. Let U he a finite field and 
Pa be a probabiUty distribution on a set of x n matrices. It should be noted that A represents a corresponding 
hnear function A:U^—f U^-^ , where we define = U^^ . 

Let t{u) be the type^ of u e W", where a type is characterized by the number nvu of occurrences of each 
symbol in the sequence u. Let W be a set of all types of length n except t(0), where is the zero vector. For 
the probabiUty distribution pA on a set of x n matrices, let S{pA,t) be defined as 

S{pA,t) = Y,Pa{A)\{u &W:Au = 0,t(n) = t}\. 

A 

For H C H, we define aAin) and f3A{n) as 

|W| S{pA,t) 

~ - — -. — • max — 

ten S{uA,t) 

(3A{n)= S{pA,t), (10) 

teH\H 

where ua denotes the uniform distribution on the set of all Ia'x n matrices. When U = GF(2) and 7i is a set 
of high-weight types, a a measures how the ensemble {A^pa) differs from the ensemble of all I a x n matrices 
with respect to high-weight part of average spectrum and (Sa provides the upper bound of the probability that 
the code {u&U^ : Au = 0} has low-weight codewords. It should be noted that 

~ / ^ _ S{pA,t) 

aA(n) = max-— — (11) 

ten t){UA,t) 

is introduced in [26][2][10][33][34] instead of aA{n). We multiply the coefficient |Im>l|/|ZY|'-^ so tiiat ola 
satisfies (H2). 
We have the following theorem. 

Theorem 1: Let {A,pa) be an ensemble of matrices and assume that pa {{A : Au = 0}) depends on u only 
through the type t{u). If |W^|/|Im^| satisfies (HI) and (aA(n), /3A(n)), defined by (9) and (10), satisfies (H2) 
and (H3), then {A,pa) has an (aA,y9^)-hash property. 
The proof is given in Section VI-B. 

Next, we consider the independent combination of two ensembles {A,pa) and {B,pb), of x n and l^xn 
matrices, respectively. We assume that {A,pa) has an {ola, y9^)-hash property, where {aAin), PA{n)) is defined 
as (9) and (10). Similarly we define (as(n),/3B(n)) for an ensemble {B,pb) and assume that {B,pb) has an 
(as,/3B)-hash property. Let pab be the joint distribution defined as 

Pab{A, B)=pa{A)pb{B). 

We have the following two lenmias. The proof is given in Section VI-B. 

' As in [6], the type is defined in terms of the empirical probability distribution In our definition, the type is the number nvu of 
occurences which is different from the empihcal probabhty distribution. 
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Lemma 7: Let (aAB('^), defined as 

OLAB{n) = aA{n)aB{n) 
0AB{n) = min{/3A(n),/3B(n)}- 
Then the ensemble [A x B,pab) of functions A^B-.W^ ^ uU+Ib defined as 

A®B{u)= {Au,Bu) 

has an (Q:AB,/3^5)-hash property. 
Lemma 8: Let {aAB^n), (3'j^g{n)) be defined as 

oeAB{n) = aA{n)aB{n) 

, _ aAin)PB{n) aBin)PA{n) , ^ , , , , 
= |W| + llmSI + W/?B(n). 

Then the ensemble {A x B,pab) of functions A(g)B -.W" x V" ^ Z^'^ x V''' defined as 

A (g) t;) = {Au, Bv) 

has an (aAs,/3AB)"hash property. 

Finally, we introduce an ensemble of g-ary sparse matrices, where the binary version of this ensemble is 
proposed in [21]. In the following, let U = GF(g') and = nR. We generate an x n matrix A with the 
following procedure, where at most r random nonzero elements are introduced in every row. 

1) Start from an all-zero matrix. 

2) For each i e {1, . . . , n}, repeat the following procedure r times: 

a) Choose (j, a) G {1, . . . , Ia} x [GF((j') \ {0}] uniformly at random. 

b) Add a to the (j, i) component of A. 

Let {A, Pa) be an ensemble corresponding to the above procedure. It is proved in Section VI-C that pA {{A : Au - 
depends on u only through the type t{u). Let (ayi('T-), be defined by (9) and (10) for this ensemble. 
We assume that column weight r = O(logn) is even. Let w{t) be the weight of type t = (<(0), . . . ,t{q— 1)) 
defined as 

9-1 

«;(t)^^t(i) 

i=l 

and be defined as 

w{u) = w{t{u)). 

We define 

n = {t: wit) > ^Ia}. (12) 

Then it is also proved in Section VI-C that 

{u e U^-^ : w{u) is even}, if g = 2 



Im^ 



U^, ifq>2 
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\ImA\ ^ 



2, if q = 2 
1, ifq>2 

and there is ^ > such that (a^, /3a) satisfies (H2) and (H3). From Theorem 1, we have the following theorem. 

Theorem 2: For the above ensemble [A^pa) of sparse matrices, let {(XAin)^ PA{n)) be defined by (9), (10), 
(12), and suitable {r(n)}^i and ^ > 0. Then {A,pa) has an (aA,/3A)-hash property. 
We prove the theorem in Section VI-C. It should be noted here that, as we can see in the proof of the theorem, 
the asymptotic behavior (convergence speed) of (a a, /3a) depends on the weight r. 

Remark 1: It is proved in [26][33] that {aA{n), PA{n)), defined by (11) and (10), satisfies weaker properties 

log a 4 (n) 

lim ^ ' = (13) 

n — ^oo Tl 

and (H3) when q = 2. It is proved in |2, Section 111, Eq. (23), (82)] that {ctAin), Pa{ti)) of another ensemble 
of Modulo-g LDPC matrices satisfies weaker properties (13) and (H3). 

V. Coding Theorems 

In this section, we present several coding theorems. We prove these theorems in Section VI based on the 
hash property. 

Throughout this section, the encoder and decoder are denoted by (p and (p~^, respectively. We assume that 
the dimension of vectors x, y, z, and w is n. 

A. Slepian-Wolf Problem 

In this section, we consider the Slepian-Wolf problem illustrated in Fig. 1. The achievable rate region for 
this problem is given by the set of encoding rate pair {Rx,Ry) satisfying 

Rx > H{X\Y) 
Ry > H{Y\X) 

Rx+Ry> H{X,Y). 

The achievabihty of the Slepian-Wolf problem is proved in [3] and [5] for the ensemble of bin-coding and all 
g-ary hnear matrices, respectively. The constructions of encoders using sparse matrices is studied in [35] [40] [34] 
and the achievability is proved in [33] by using ML decoding. The aim of this section is to demonstrate the 
proof of the coding theorem based on the hash property. The proof is given in Section VI-D. 

We fix functions 

which are available to construct encoders and a decoder. We define the encoders and the decoder (illustrated 
in Fig. 6) 

ipx-.X""^ X^^ 
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Encoders 



X 
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B 



Decoder 



Ax 
By 



9AB 



Ax 
By 

{x,y) 



Fig. 6. Construction of Slepian-Wolf Source Code 



as 



where 



ipx{x) = Ax 
Wiy) = By 
(p~'^{bx,bY) = gAB{bx,bY), 



gAB{bx,bY) = a,vg max uxr(a;',y')- 

(x' ) 6Ca (fex ) X Cb (by ) 

The encoding rate pair {Rx,Ry) is given by 

_Ulog^ 

n 

_ ^giog|y| 

ity = 

n 

and the error probabiUty ErrorxF(^, B) is given by 

Errorxr(^,-B) = Hxy {{{x,y) : ip~\(px{x),ipY{y)) ^ {x,y)}) • 

We have the following theorem. It should be noted that and alphabets X and y may not be binary and the 
correlation of the two sources may not be synmietric. 

Theorem 3: Assume that {A,pa), {B,Pb), and {A xB,pa'x Pb) have hash property. Let {X, Y) be a pair 
of stationary memoryless sources. If {Rx,Ry) satisfies 

Rx > H{X\Y) (14) 

Ry > H{Y\X) (15) 

Rx + Ry > H{X,Y), (16) 

then for any S > and all sufficiently large n there are functions (sparse matrices) A G A and B G B such 
that 

Etvotxy{A,B) < S. 
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R < I{X; Y) 



Fig. 7. Channel Coding 



Remark 2: In [3][5], random (linear) bin-coding is used to prove the achievabUity of the above theorem. 
In fact, random bin-coding is equivalent to a uniform ensemble on a set of all (Unear) functions and it has a 
(1, 0)-hash property. 

Remark 3: The above theorem includes the fixed-rate coding of a single source X as a special case of the 
Slepian-Wolf problem with |3^| = 1. This implies that the encoding rate can achieve the entropy of a source. 
It should be noted that source coding using a class of hash functions is studied in [22, Section 14.2][17]. 

Remark 4: Assuming |3^| = 1, we can prove the coding theorem for a charmel with additive noise X by 
letting A and {x : Ax = 0} be a parity check matrix and a set of codewords (channel inputs), respectively. 
This impUes that the encoding rate of the channel can achieve the channel capacity. The coding theorem for a 
channel with additive noise is proved by using a low density parity check (LDPC) matrix in [26] [2] [10]. 

B. Gel'fand-Pinsker Problem 

In this section we consider the Gel'fand-Pinsker problem illustrated in Fig. 2. 

First, we construct a code for the standard channel coding problem illustrated in Fig. 7, which is a special 
case of Gel'fand-Pinsker problem. 

A channel is given by the conditional probabiUty distribution Hy\x^ where X and Y are random variables 
corresponding to the channel input and channel output, respectively. The capacity of a channel is given by 

Capacity = max/(X; Y), 

MX 

where the maximum is taken over all probability distributions fix and the joint distribution of random variable 
{X, Y) is given by 

f^xY{x,y) = HY\x{y\x)nx{x). 
The code for this problem is given below (illustrated in Fig. 7). We fix functions 

A : Af" ^ X^^ 

B : X" X^" 

and a vector c e X^^ available to construct an encoder and a decoder, where 

_ n[H{X\Y)+eA] 
log\X\ 

_ n[I{X;Y)-eB] 

We define the encoder and the decoder 
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Fig. 8. Construction of Channel Code 



as 



(/?(m) = gAB{c,m) 
V~^{m) = BgA{c\y), 

where 

fifAs(c,m) = arg max lJ.x{x') 

x' GCAB{c,m) 

9A{c\y) = arg max nxY{x'\y). 
Let M be the random variable corresponding to the message m, where the probabihty pMim) is given by 

ifmelmB 



PM{m) = < 
The rate of this code is given by 



R{B) 



0, if m ^ ImB. 

log |Im^| 



n n 

and the decoding error probability Errory-|x(^, B, c) is given by 

ETrorY\xiAB,c) = ^pMim)iiY\x{yW{m))xi(p^^iy) ^m). 

m,y 

In the following, we provide an intuitive interpretation of the construction of the code, which is illustrated 
in Fig. 8. Assume that c is shared by the encoder and the decoder. For c and a message m, the function qab 
generates a typical sequence x G Tx,^ as a channel input. The decoder reproduces the channel input x by 
using qa from c and a channel output y. Since {x, y) is jointly typical and Bx = m, the decoding succeed if 
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the amount of information of c is greater than H{X\Y) to satisfy the collision-resistant property. On the other 
hand, the total rate of c and m should be less than H{X) to satisfy the saturating property. Then we can set 
the encoding rate of m close to H{X) - H{X\Y) = I{X; Y). 

We have the following theorem It should be noted that alphabets X and y are allowed to be non-binary, and 
the channel is allowed to be asymmetric. 

Theorem 4: For given eg > satisfying 

£B-EA< a/6 [eg - e^]log \X\ < sa, 

assume that {A,pa) and {A x B,pA x pb) have hash property. Let ij,y\x be the conditional probability 
distribution of a stationary memoryless channel. Then, for all 5 > and sufficiently large n there are functions 
(sparse matrices) A e A, B e B, and a vector c e ImA such that 

R{B) > I{X- Y)-eB-5 
Errory|x(^, -B,c) < 8. 

By assuming that jix attains the channel capacity and J ^ 0, ^ 0, the rate of the proposed code is close 
to the capaticy. 

Next, we consider the Gel'fand-Pinsker problem illustrated in Fig. 2. A channel with side information is given 
by the conditional probability distribution fJbY\xzy where X, Y and Z are random variables corresponding to 
the channel input, channel output, and channel side information, respectively. The capacity of a channel with 
side information is given by 

Capacity = max [I{W;Y) - I(W;Z)], 

where the maximum is taken over all conditional probability distributions lJ,xw\z and the joint distribution of 
random variable (X, Y, Z, W) is given by 

IJ.xYzw{x-, y, z, w) = nxw\z{x, w\z)iJ.Y\xz{y\x, z)iiz{z). (17) 
In the following, we assume that Hxw\z is fixed. We fix functions 



A: 


;W"- 




B ■ 


:W"- 




A: 


; Af" - 





and vectors c € W and c G X'-a available to construct an encoder and a decoder, where 

_ n[H{W\Y)+sj^\ 
^ log|W| 

_ n[H{W\Z) - H{W\Y) - gg] 
^~ log|W| 

^ n[I{W;Y)-I{W;Z)-eB] 
log|W| 

_ n[H{X\Z,W)-e^] 
^~ \og\X\ 
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Fig. 9. Construction of Gel'fand-Pinsker Channel Code 



We define the encoder and the decoder 



as 



^-1 : yr^ _ w'b 

V3(m|z) = g^{c\z,gAB{c,m\z)) 
ip~'^{m) = BgA{c\y), 



where 



gAB{c, m\z) = a.Yg max nwizi^'lz) 
fif^(c|z, w) = arg max /^xizivla^l^, w) 
5A(c|y) = arg max av^vY{w'\y). 

Let M be the random variable corresponding to the message m, where the probabihty pm {m) and pmz {m, z) 
are given by 



PM{m) 



0, if m ^ ImB 
PMz(m,z) = pM{m)iJ,z{z). 
The rate -R(-B) of this code is given by 

R^B) ^ i^^M 
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n n 

and the decoding error probability Errory-|xz(^i B, A, c, c) is given by 

Errory|jc2(A,S, A, c,c) = ^ pM{m)iiz{z)iiY\xz{yWi'm,z),z)x{'fi~^{y) ¥= rn). 

m,y,z 

In the following, we provide an intuitive interpretation of the construction of the code, which is illustrated in 
Fig. 9. Assume that c is shared by the encoder and the decoder. For c, a message m, and a side information z, 
the function qab generates a typical sequence w G Tz^-y{z) and the function generates a typical sequence 
X e 7^nyx,7('"') z) as a channel input. The decoder reproduces the channel input w by using qa from c and 
a channel output y. Since {w, y) is jointly typical and Bw = m, the decoding succeed if the rate of c is 
greater than H{W\Y) to satisfy the colUsion-resistant property. On the other hand, the rate of c should be less 
than H{X\Z, W) and the total rate of c and m should be less than H{W\Z) to satisfy the saturating property. 
Then we can set the encoding rate of m close to H{W\Z) - H{W\Y) = I{W; Y) - I{W; Z). 

We have the following theorem. It should be noted that alphabets X, y, W, and Z are allowed to be 
non-binary, and the channel is allowed to be asymmetric. 

Theorem 5: For given £^,£b,£^ > satisfying 

£B-eA< v/6[£K - eA] log \Z\\W\ < ea (18) 

KywiQsj) < £A, (19) 

assume that {A, pa), {A xB,pa'xpb) and {A, p^ have hash property. Let Hy\xz be the conditional probabiUty 
distribution of a stationary memoryless channel. Then, for all 5 > and sufficiently large n there are functions 
(sparse matrices) A G A, B G B, A G A, and vectors c G ImA, c G ImA such that 

R{B) > I{W; Y) ~ I{W; Z) - bb - 5 
EriorY\x z{A,B, A, c,c) < 6. 

By assuming that Hxw\z attains the Gel'fand-Pinsker bound, and ^ — > 0, eg — * 0, the rate of the proposed 
code is close to this bound. 

The proof is given in Section VI-E. It should be noted that Theorem 4 is a special case of Gel'fand-Pinsker 
problem with \Z\ = 1 and W = X. 

Remark 5: In [24], the code for the Gel'fand-Pinsker problem is proposed by using a combination of two 
sparse matrices when all the alphabets are binary and the channel side information and noise are additive. In 
their constructed encoder, they obtain a vector called the 'middle layer' by using one of two matrices and 
obtain a channel input by operating another matrix on the middle layer and adding the side information. In 
our construction, we obtain w by using two matrices A, B, and qab, where the dimension of w differs from 
that of the middle layer. We obtain channel input x by using A and instead of adding the side information. 
It should be noted that our approach is based on the construction of the channel code presented in [33][29], 
which is also different from the construction presented in [12] [2]. 
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Fig. 10. Lossy Source Coding 
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Fig. 11. Construction of Lossy Source Code 



C. Wyner-Ziv Problem 

In this section we consider the Wyner-Ziv problem introduced in [47] (illustrated in Fig. 3). 
First, we construct a code for the standard lossy source coding problem illustrated in Fig. 10, which is a 
special case of the Wyner-Ziv problem. 

Let p : A" X 3^ ^ [0, oo) be the distortion measure satisfying 

Pmax = maxp(a;,t/) < oo. 

We define pn{x,y) as 

n 

pn{x,y) = ^p{xi,yi) 

for each x = {xi,. . . , a;„) and y = {yi, . . . , yn). For a probability distribution nx, the rate-distortion function 
Rx{D) is given by 

Rx{D)= min I{X-Y), 

Exy[p(X,Y)]<D 

where the minimum is taken over all conditional probability distributions iiy\x and the joint distribution ijlxy 
of {X, Y) is given by 

HxY{x,y) = px{x)pY\x{y\x). 
The code for this problem is given in the following (illustrated in Fig. 11). We fix functions 
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and a vector c e 3^'-^ available to construct an encoder and a decoder, where 

_ n[g(y|X) - sa] 

^- \og\y\ 

_ n[I{Y;X)+eB\ 

^- \og\y\ ■ 

We define the encoder and the decoder 

: A*" ^ y« 

as 

(p(a;) = BgA{c\x) 
V>~^{b) = gAB{c,b), 

where 

SfA(c|a;) = arg max ^i-|jc(y'|a;) 
5fAs(c,6) = arg max fiyiv')- 

y'eCAB{c,b) 

In the following, we provide an intuitive interpretation of the construction of the code, which is illustrated 
in Fig. 11. Assume that c is shared by the encoder and the decoder For c and x, the function gA generates y 
such that Ay = c and {x,y) is a jointly typical sequence. The rate of c should be less than H{Y\X) to satisfy 
the saturation property. Then the encoder obtains the codeword By. The decoder obtains the reproduction y 
by using gAB from c and the codeword By if the rate of c and By is greater than H{Y) to satisfy the 
collision-resistant property. Then we can set the encoding rate close to H(Y) — H{Y\X) = I[X;Y). Since 
{x,y) is jointly typical, pn{x,y) is close to the distortion criterion. 

We have the following theorem. It should be noted that a source is allowed to be non-binary and unbiased 
and the distortion measure p is arbitrary. 

Theorem 6: For given ea >,sb > satisfying 

assume that {A,pa) and {B,pb) have hash property. Let X be a stationary memory less source. Then for all 
sufficiently large n there are functions (sparse matrices) A & A, B & B, and a vector c e Im^ such that 

R{B) = I{X-Y) + SB 

< ExY [p{X,Y)\ + 2,\X\\y\p^^^JeA- 

n 

By assuming that Py\x attain the rate-distortion bound and by letting £a,£b 0, the rate-distortion pair of 
the proposed code is close to this bound. 
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Fig. 12. Construction of Wyner-Ziv Source Code 



Next, we consider the Wyner-Ziv problem introduced in [47] (illustrated in Fig. 3). Let p : A" x W ^ [0, oo) 
be the distortion measure satisfying 



We define pn{x,w) as 



Pmax = maxp(a;,-w;) < oo. 



Pn{x,w) = ^p{xi,Wi) 



for each x = [xi, . . . ,Xn) and w = . . . , w„). For a probability distribution jj^xz, the rate-distortion 
function Rx\z{D) is given by 

Rx\z{D) = min [I{X; Y) - I{Y; Z)] , 

ExYzlp{XJ{Y,Z))]<D 

where the minimum is taken over all conditional probability distributions Hy\x and functions f -.y x Z 
and the joint distribution p,xYZ of {X, Y, Z) is given by 

P^XYzix, y, z) = nxzix, z)iJ,Y\xiy\x). (20) 

In the following, we assume that ij.y\x is fixed. We fix functions 

B-.y ^ y^" 

and a vector c G y''-^ available to construct an encoder and a decoder, where 

n[H{Y\X) - sa] 



\og\y\ 



_ n[H{Y\Z) - H{Y\X) + eb] 

^- \og\y\ 

_ n[I{X-Y)-I{Y-Z)-^eB\ 
log 13^1 
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We define the encoders and the decoder (illustrated in Fig. 12) 

: Af" ^ 

as 

ip{x) = BgA{c\x) 
<^"^(6|z) = fnigAB{c,b\z),z) 

where 

gA{c\x) = a.vg max iJ.Y\x{y'\x) 

y'eCAic) 

QAsic, b\z) = a.vg max t^Y\z{y'\z) 
y'eCAB{c,b) 

and we define z) = {wi, . . . , Wn) by 

Wi = f{yi,Zi) 

for each y = {yi, ... ,yn) and z = {zi, ... , Zn). 
The rate R{B) of this code is given by 

RiB) . ^^1^. 
n 

In the following, we provide an intuitive interpretation of the construction of the code, which is illustrated 
in Fig. 12. Assume that c is shared by the encoder and the decoder. For c and x, the function qa generates y 
such that Ay = c and {x, y) is a jointly typical sequence. The rate of c should be less than H{y\X) to satisfy 
the saturation property. Then the encoder obtains the codeword By. The decoder obtains the reproduction y by 
using qab from c, the codeword By, and the side information z if the rate of c and By is greater than H{Y\Z) 
to satisfy the collision-resistant property. Then we can set the encoding rate close to H{Y\Z) — H{Y\X) = 
I{X; Y) — I{Y] Z). Since {x, y, z) is jointly typical, Pn{x, f{y, z)) is close to the distortion criterion. 

We have the following theorem. It should be noted that a source is allowed to be non-binary and unbiased, 
side information is allowd to be asymmetric, and the distortion measure p is arbitrary. 

Theorem 7: For given > satisfying 

eA + Kyz{'ieA)<ei3, (21) 

assume that {A,pa) and (B,pb) have hash property. Let (X, Z) be a pair of stationary memoryless sources. 
Then for all sufficiently large n there are functions (sparse matrices) A e A, B e B, and a vector c e Im^ 
such that 

R{B)=I{X;Y)-I{Y;Z)+SB 
-^^ < ExYZ [p{X,f{Y,Z))\ SjA'liyilZlpmaxVe.A- 
By assuming that Py\x 'Uid / attain the Wyner-Ziv bound and letting £a,£b 0, the rate-distortion pair of 
the proposed code is close to this bound. 
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Fig. 13. Construction of One-helps-one Source Code 



The proof is given in Section VI-F. It should be noted that Theorem 6 is a special case of the Wyner-Ziv 
problem with \Z\ = l,W = y, and /(?/, z) = y. 

Remark 6: In [25][36][23][27][14], the lossy source code is proposed using sparse matrices for the binary 
alphabet and Hamming distance. In their constructed encoder proposed in [36][23][27][14], they obtain a 
codeword vector called the 'middle layer' (see [23]) by using a matrix. In their constructed decoder, they 
operate another matrix on the codeword vector. In our construction of the decoder, we obtain the reproduction 
y by using a sparse matrix A and gA and compress y with another matrix E. It should be noted that the 
dimension of y is different from that of the middle layer and we need ML decoder qab in the construction of 
the decoder, because y is compressed by using E. In [24], the code for the Wyner-Ziv problem is proposed 
and there are similar differences. Our approach is based on the code presented in [28] [30] and similar to the 
code presented in [42] [43] [49]. 

D. One-helps-one Problem 

In this section, we consider the One-helps-one problem illustrated in Fig. 4. The achievable rate region for 
this problem is given by a set of encoding rate pair {Rx,Ry) satisfying 

Rx > H{X\Z) 
Ry>I{Y;Z), 

where the joint distribution fixYZ is given by 

IJ,xYz{x,y,z) = iJLxY{x,y)iiz\Y{z\y)- (22) 

In the following, we construct a code by combining a Slepian-Wolf code and a lossy source code. We assume 
that iiz\Y is fixed. We fix functions 

B-.X"^ X^B 
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26 



and a vector c e Z^^ available to construct an encoder and a decoder, where 



Ia = 
Ib = 



loglA"! 




n[H{Z\Y) - 


- sa] 


log|Z| 




n[I{Y;Z) + 


■£b] 


log 1^1 



We define the encoders and the decoder (illustrated in Fig. 13) 

ip-'^ : Af'e X Z'" Af" 



as 



(px{x) = Bx 
ifriy) = BgA{c\y) 
ip~'^{bx,bY) = gg{bx,gAB{c,bY)), 



where 



9A{c\y) = arg max uz\Yiz'\y) 
z'eCA(c) 

5ab(c, by) = arg max Hz{z') 

z'eCAB{c,bY) 

g§{bx\z) = aig max fj,x\z{x'\z). 

x'eCg{bxj 

The pair of encoding rates {Rx,Ry) is given by 

sty 



R 



Y 



n 

lBl0g\Z\ 



n 

and the decoding error probability Errorxy(^, B, B, c) is given by 

Etj:otxy{A,B,B,c) = iixy {{{x,y) : (fi~'^{ipx{x),(pY{y)) ^ x}) . 

We have the following theorem. 

Theorem 8: For given e^, sb, > satisfying 

£B > £a + (z{3£a) (23) 
eg > 2Cxz{SeA), (24) 
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assume that (^,^4), (B.pb), and {13, pg) have hash property. Let {X,Y) be a pair of stationary memoryless 
sources. Then, for any 5 > and and all sufficiently large n, there are functions (sparse matrices) A G A, 
B G B, 13 G 13, and a vector c G Im^ such that 

Rx=H{X\Z) + eg 

Ry = I{X- Z) + EB 

Errorjfy(A,5,5,c) < 5. 

The proof is given in Section VI-G. 

VI. Proof of Lemmas and Theorems 

In the proof, we use the method of types, which is given in Appendix. Throughout this section, we assume 
that the probability distributions of pc, pq, Pm are uniform and the random variables A, B, A, B, C, C and 
M are mutually independent. 

A. Proof of Lemmas 1-6 

We prepare the following two lemmas, which come from the fact that pc is the uniform distribution on 1mA. 
and random variables A and C are mutually independent. 

Lemma 9: Let pa be the distribution on the set of functions and pc be the uniform distribution on Im^. 
We assume that a joint distribution pAc satisfies 

PAc{A,c) =pa{A)pc{c) 

for any A and c e ImA. Then 

J2pc{c)x{Au = c) = (25) 



for any A and uGlA^, 



for any u € U", and 



llm^l 

J2pac{A, c)x{Au = c) = (26) 



^PAc{A,c)\gnCA{c)\ = J2J2PAc{A,c)xiAu = c) 

A,c ueQ A,c 

\G\ 



(27) 



llmy^l 



PAC {{{A, c):gn Ca{c) ^ 0}) < J|L (28) 

for any C W. 

Proof: First, we prove (25). Since Au is determined uniquely, we have 

Y^x{Au = c) = 1. 

c 

Then we have 

x{Au = c) 



X]^'C'(c)x(^w = c) = ^ 



llm^l 



March 19, 2009 



DRAFT 



\lmA\ ■ 

Next, we prove (26). From (25), we have 

J2pac{A, c)x{Au = c) = ^pa(>1) ^Pc(c)x(>1m = c) 

A,c A c 

1 

~ llm^l ■ 

Next, we prove (27). From (26), we have 

^PAc{A,c) \gnCA{c)\= ^pAc{A,c)^x{Au = c) 

A,c A,c ueQ 

= X] ^PAc{A, c)x{Au = c) 
ueQ A,c 

= y^ 



llm^l ■ 

Finally, we prove (28). From (27), we have 

PAC {{{A, c):gn Ca{c) ^ 0}) = PAC {{{A, c) : 3u e 5 n Ca{c)}) 

< Y.PAc{{{A,c):ueCA{c)}) 

ueg 

= ^PAc{A,c)x{Au = c) 
ueQ A,c 

\lmA\ ■ 

Proof of Lemma 1: Since {A,pa) has an (aA,/3A)"hash property, we have 

PA{{A:[g\{u}]r\CA{Au)^^)< Pa{{A:Au = Au'}) 

u'ee\{u} 

< \{u} n [g \ {u}] I + ^^\l2Ar ^ ^^^^l^""^'' \ ^""^l^^^ 

Proof of Lemma 2: First, since has an (aA,/3A)"hash property, we have 

X! Z^^'^cl^, c)x(^M = c)x(Am' = c) = ^ ^pAc(>l, c)x(^w = c)x{Au = Aw') 

u.u'GT a c 



^ E T.P^{A)xiAu = Au') 
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^ |T| \T\^cxA \T\Pa 
- \ImA\ |Im^|2 ^ |Im^| ' 



(29) 



where the third equality comes from (25). 
Next, we have 



^PAciA C) 
A,c 

= ^PAciA C) 



ITI 



|Im^| 



.ueT 



2\T\ ^ X V- X 1^1' 



< 



w,u'eT A,c 

|Tp[a^-l] , |T|[/3^ + l] 



■ueT A,c 



|Im^|2 



" llm^l' llni^l ' 

where the last inequality comes from (27) and (29). 
Finally, from the fact that T we have 

PAC {{{A, c):Tn Ca{c) = 0}) = pac {{{A, c) : Vw e T, Au ^ c}) 



(30) 



= PAC |^|(Ac) : xiAu = c) = O^j 



<PAc{{ (A, c) : 



E ^(^^ = 



iri 



|Im^| 



- \ImA\j^ 



< 



J2a,oPac{A, c) [E^gt xiAu = c) 



1^1 

|Imyt| 



< 



llm^P 

|rp[a^-i] |r|[/3^+i] 

|Im^P ^ |Im^| 

|W|[/3a + 1] 
= "A - IH , 

where the second inequaUty comes from the Markov inequality and the third inequaUty comes from (30). 
Proof of Lemma 3: Since {A,pa) has an (a^, /3^)-hash property, we have 







gnCA{c)^(l) 




/ 


f 


gr\CA{Au 


PAC 


< 


{A,c): 


>j ^PAC 


< 


(Ac) 








u e Ca{c) 




\ 




u e Ca(c) 



< 



: J2PAiA)xiG n Ca{Au) ^ ID)J2pc{c)x{Au = c) 

A c 

PA ({A:gnCA(Atx)^0}) 
\lmA\ 

\G\aA , /3a 



|Im^|2 llmy^l' 

where the second equality comes from the fact that random variables A and C are independent, the third equality 
comes from (25), and the inequahty comes form Lemma 1. ■ 
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Proof of Lemma 4: By applying Lemma 1 to the set [G \ {ua,c}] (^Ca{c), we have 

\[g\{uA,o}]nCA{c)\aB 



PABC {{{A, B, c):[g\ {uA,c}] n Cab{c, Bua,c) ¥= 0}) < J2p^c{A, c) 

A,c 

\g\aB 



\lmB\ 
\gnCA{c)\aB 



llmBI 



|Im^||ImB| 



where the last equahty comes from (27). 
Proof of Lemma 5: Let 



g{v) = {u : f^u\viu\v) > 2-"[«(^l^)-2s]^ 
If g{v) n Ca(c) = and T(t;) n Ca(c) ^ 0, then there is u e T(v) n Ca(c) and gA{c\v) satisfies 

„uiv{9a{c\v)\v) < 2-"[^(^^l^)-2el. 

Since u G Ca(c), we have 

MC/|y(fl'A(c|w)|v) > /i,7|y(u|i;). 

This impUes that gA{c\v) G T{v) from the assumption of T{v). From Lenmia 2 and (28), we have 

PAC {{{A, c) : gA{c\v) ^ T(t;)}) < 1 - p^c {{{A, c) : ffA(c|t;) € T (t;)}) 



< 1 - PAC < (A, c) : 



T(i;)nCA(c)7^0 



g{v)nCA{c) = 9 

< PAC {{{A, c) : T{v) n Ca(c) = 0}) + pAc {{{A, c) : g{v) D Ca(c) ^ 0}) 

|W|[/3a + 1] , 

|T(t;)| |Im^| 

-ne|7 /|(^ 



^ , IWI [/3a + 1] , 2-"^|^^|'^ 
where the last inequality comes from the fact that 



Proof of Lemma 6: When T{v) fl Ca(c) ^ 0, we can always find the member of T{v) by using qa- From 
Lemma 2, we have 

PAC {{{A, c) : gA{c\v) i T{v)\) < pac {{{A, c) : T{v) n Ca(c) = 0}) 
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B. Proof of Theorem 1 and Lemmas 7 and 8 
For a type t, let Ct be defined as 

Ct = {li e ZY" : t{u) = t} . 

We assume that pa {{A : Au = 0}) depends on u only through the type t{u). For a given u e Ct, we define 

UA,t = UA {{A : Au = 0}) 

PA,t = PA {{A : Au = 0}) , 

where ua denotes the uniform distribution on the set of all Ia'x n matrices and we omit u from the left hand 
side because the probabilities ua {{A : Au = 0}) and pa {{A : Au = 0}) depend on u G Ct only through the 
type t. 

We use the following lemma in the proof. 
Lemma 10: 

aA{n) = \ImA\ maxpA,t (31) 
ten 

Mn)= J2 \Ct\PA,t, (32) 

where W is a set of all types of length n except the type of the zero vector. 

Proof: Since we can find matrices A to satisfy Au = for u e Ct, we have 

S{pA,t) = ^PA{A) J2 1 
A ueCt 

Au=0 

ueCt A:Au=0 
= \Ct\PA,t- 

S{UA,t) = \Ct\UA,t- 



We have 



Similarly, we have 



The lemma can be shown immediately from (9), (10), and the above equalities. 

Proof of Theorem 1: Without loss of generality, we can assume that |T| < |T'|. We have 

^ PA {{A : Au = Au'}) = PA {{A : A[u - u'] = 0}) 
u'eT' it'eT' 

< Pa{{A:AO = Q}) + Y E 

mgt nT' ten ueT 

u'eT 

t{u — U )=:t 
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^ E 1 + E E E Eic*K* 

u'eT' 

t{u—u')=t 

<\TnTuT. E m + m E ^(^-*) 

u'eT' ^ ^ 
t{u-u')e'H 

,,rnrHffl».im,„, 

= \Tnr\ + '^"^^Ai +-in{|T|,|T'|)/;^(n). 

where the third inequality comes from (31) and the last equahty comes from the assumption |T| < \T'\. Since 
{cxa,0a) satisfies (H2) and (H3), we have the fact that {A,pa) has an (aA,/3A)"hash property. ■ 
Proof of Lemma 7: Without loss of generality, we can assume that |T| < |T'| and I3a{ti) < /3s(n). Similar 
to the proof of Theorem 1, we have 



J2 Pab{{{A,B) ; iAu,Bu) = iAu',Bu')}) 
u'eT' 

= Yl PA{{A:A0 = 0})pBi{B:B0 = 0}) + J2 E PA,tPB,t 
ueTnT' teH ueT 

u'eT' 

t{u—u')=t 

ten "e^, tew\-H 
t{u-u')e'H 

where the first inequality comes from the fact thatps^t < 1- Since {aAB{n) , PAB{n)) satisfies (H2) and (H3), 
{A X B,pab) has an (a^s, /3AB)-hash property. ■ 
Proof of Lemma 8: Without loss of generality, we can assume that |T| < |T'|. Let Hu and Tiy be defined 
similarly to the definition of H, and Hu and Hy be defined similarly to the definition of H. Similar to the 
proof of Theorem 1, we have 



J2 PAB {{{A, B) : {Au, Bv) = {Au', Bv')}) 



{u',v')eT' 



J2 Pa{{A:AO = 0})pb{{B:BO = 0})+ J2 E PA,tuPB,t^ 

(u,t))eTnT' tueHu {u,v)eT 

tv^Hv (u',v')eT' 
t(u-u')=tu 
t{y — v')—t\? 
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Si 



■sp aA{n)aB{n) \^ \^ \^ \^ aA{n)pB,tv 
- ^ ^ ^ \'ho.A\\lmB\ ^ 2^ 2^ 2^ \ho.A\ 

(■u.,v)eTnT' tu£Hu (■>f,-")(^T^ tuGHu tveHvXHv i'^'-")^'^ {u' ,v')eT' 
tv^Hv {u',v')eT' t{u-u')=t, 
t{u—u')=tu 
t{v—v')=tv 

+ E E E E E E E ^-."^B.v 

t{u — u')—tn t{u — u')—ti4 

t{v — v')—ty t{y—v')=t\> 

^ E 1+ E E iwiiiinBi + E ^ E ic*vKtv 

t{u—u')=tu 
t{v—v^)=t\> 

+ E E \<^tu\PA,tu+ E E E l^tvlPs.tv 



t{v—v')=tv 



— /, \T\\T'\aA(n)afi(n) 



|W| + |Img| + /^4")/^B(n) 



where the first inequahty comes from the fact that PA,tu ^ 1 PB,tv 
and (H3), (>l x B,pab) has an (aAB,/3AB )"hash property. 



< 



1 . Since (aASj/S'^s) satisfies (H2) 



C. Proof of Theorem 2 

Throughout this section, let U = GF(g), I = nR, and pa be an ensemble of Z x n sparse matrices as 
specified in Section IV, where we omit dependence on the ensemble of /. It should be noted that / ^ oo by 
letting n ^ oo. 

First, we prepare lemmas that provide the analytic expression of PA,t- 

Lemma 11: We consider a random-walk on GF(g'') defined as the following. Let Cn € GF(g') be the position 
after n steps. At each unit step, the position is renewed in the following rule. 

1) Choose {i, u) e {1, ...,/} x GF(g) uniformly at random. 

2) Add u to the i-th element of c„. 

Then, the probability P„(c) of the position c after n steps starting from the zero vector is described by 



fc=0 



1 



qk 



[q-l]l. 



E 

k'=0 



w(c)\ fl — w{c) 



k' 



k-k' 



/ (-ir (9-1) 



k-k' 



(33) 



Proof Let C C GF(g') be defined as 



C = < 



(0,...,0,c,0,...,0) 



b-i] 



[i-i] 



J e {!,... ,1} 

c e GF(g) 



Then the transition rule of this random walk is equivalent to the following. 

1) Choose c e C uniformly at random. 

2) Add c to Cn, that is, 

Cn+l = Cn + C. 
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We have the following recursion formula for P„(c). 



if c e C, 



[q-l]V 

0, otherwise. 



P„+l(c) = Pn{c')Pl{c - C') = [P„ * Pi](c), 

c'eGF(gO 

where P„ * Pi denotes the convolution. We have (33) by using the following formulas 

dPn = [dPn^i][m] = ■■■ = [dPir 

where 'S is the discrete Fourier transform and is its inverse. 

Lemma 12: The probability pa {{A : Au = 0}) depends on u only through the type t{u), that is, if w{t) 
w{t') then pA,t = PA,t'- Furthermore, 



PA,t 



qk 



= -y 

^ fc=0 

Proof: For u= (ui, . . . , we define u* = . . . , u*) as 

1, ifuij^O 



0, if Ui = 0. 

Similarly as in the proof of [10, Lemma 1], we can prove that two sets {A : Au = 0} and {A : Au* = 0} 
are in one-to-one correspondence. Then we have 

PA {{A : Au = 0}) = PA {{A : Au* = 0}) , 

that is. Pa {{A : Au = 0}) depends on u only through w{t). 

Since pa {{A : Au* = 0}) is equal to the probabihty that the position of the random walk defined in Lennma 
11 starts from the zero vector and returns to the zero vector after w{u*)t steps, we have 



PA,t = Pwit)r{0) 



1 



qk 



[q-l]l. 



U{t)7 



iq-l)K 



Next, we prove the following lemma. 

Lemma 13: If the column weight r is even, then 

{u&U^ : w{u) is even}, if g' = 2 



Im>l = < 



which impUes 
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Proof: Let a^.j be the (i, j) element of A. 
First, we assume that q = 2. Then it is sufficient to prove that w{Au) is even for any possible A and u 
because 

n 



E 1- E i = E 



iu(c) is even iu(c) is odd 



w=0 



= 



which implies that |Im^| = \U\''/2. Without loss of generality, we can assume that w{u) = w and u = 
(1, . . . , 1, 0, . . . , 0). Let ttj = (aj,i, . . . , Since every column vecotor has an even weight, we have the 

fact that Erir^ w{ai) is even. In addition, we have 

w{u) 



^ w{ai) = ^ w{ai) - ^ w{ai). 

i: i=l i: 

tu(oi) is odd w{ai) is even 

This implies that the number of odd-weight vectors Oj is even because the right hand side of the above equality 
is even. Since w{Au) is a number of odd-weight vectors a^, we have the fact that 'w{Au) is even for any A 
and u e UK 

Next, we assume that q> 2. It is sufficient to prove that, for any c = (ci, . . . , c;) gUK there is A generated 
by the scheme and u G such that Au = c. This fact implies that ImA = UK Let u = (1, . . . , 1). It is 
possible to generate A satisfying 

2a, if i = j 
0, ifi^j, 

where a G GF(g) is arbitrary. Since g > 3, we have Au = c by letting a = Ci/2. ■ 
Finally we prove that {(Xaj^a) satisfies (H2) and (H3). We define the function h as 

h{0)^-eiogM-[^-0]^og,{i-0), 

where e is the base of the natural logarithm. We use the following lemmas to derive the asymptotic behavior 

of {(Xa,(3a)- 

Lemma 14: Let a be a real number. Then 



If a < -loge(/ - 1), then 



Lemma 15: 



max [h(9) + aO] < log, (1 + e") . 

O<0<1 J _ oe \ / 



max \h(0) + a9]<h(l] 
i/i<e<i \l J I 



lh{j]<l+ log, I. 



Lemma 16: 



Proof: Since 



E 

fc=i 



1 _ ^ 



2k 



l\ ^ f 2kwT\ fl 

^2E-p(-^j 

fe=l ^ ^ ^ 



1 2[Z - fc] 



i-kr 
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then we have 



E 

k=l 



2k 



fe=i 
< 2 ^ exp 



2k 



2ki 



I J \k 



where the inequahty comes from the fact that 2k /I < 1. 
Lemma 17: 



E 



1 - 



qk 



[q-l]l 



Proof: We can show the lenmia from the fact that 

qk 



[q-l]l 



< 1 



when k <[q— l]l/q and 



[q-l]l 



q[k -l] + l 
[q-l]l 



< 



[q-l]l 
= [9-1]-' 

when [q - l]l/q < k <l. ■ 

Let T be the parameter given in the procedure used for generating a sparse matrix. We assume that r and ^ 
satisfy 



r = 2 



loge 



R 



Then we have 



^|-^+^ioge(g-i)< 3- 



^r>3\ogJ 



(35) 
(36) 

(37) 



for all sufficiently large I. 

Now we are in position to prove the following two lemmas which provides the proof of Theorem 2. 
Lemma 18: 

lim aA{n) = 1. 



Proof: In the following, we first show that 



lim y 



exp - 



fe=i 



[q - l]l ) \k 



[9-l]' = 



(38) 
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il 



for all g > 2 and w > ^Z. By assuming w > £,1, we have 



< / ^ max ^ exp {-^ItO + lh{e) + I0log^{q - 1)) 
<l max exp(;[/i(6') + [log,(g-l)-eT]6']) 

1/i 1 



< Zexp I I 



^ 1\ ^ loge(9 - 1) - 



< exp (1 + logg Z + logg(g - 1) - + log^ 

< exp (-^T + 2 log, I + log, [9 - l]e) , 

where the fifth inequality comes from (37) and Lemma 14, and the sixth inequality comes from Lemma 15. 
Hence we have (38) for all g > 2 and w > ^l. 

Next, we show the lemma by assumimg that q = 2. From Lemma 16, (38), and the fact that wt is even, we 
have 



l-l r 



lim max } 



1 



2k 

T 



kl I— too W>(,1' 



l-l 

lim max > 1 

fe=l 

< 2 lim max 



2fc 



il exp (- 

k=l ^ 



2kwT\ // 

~r) U 



= 0. 



From (31) and Lemma 13, we have 



1 ' 

lim aA{n) — lim — max ^ 



fe=0 



1 '"^ 

= 1 + - lim max > 

2 n—>oo w>l^ ^ — ' 



fe=l 



2k 



1 



1. 



Finally, we show the lemma by assuming that q > 2. From (38), the first term on the right hand side of 
(34) vanishes by letting I — > oo. Since [q — l]l/q > 1/2, then the second term on the right hand side of (34) 
is evaluated by 



E 



k — WT 



< I 



I 



9-1 



[9-1] 



l — WT 



< I exp I Ih 



[q - 1]'-"'^ 

< I exp (/ logg eq - WT logg(g - 1)) 

< exp {-l[^T\og^{q - 1) - lege eq - log, I]) 
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where the third inequality comes from h{6) < 1. From q > 2 and (37), the second term on the right hand side 
of (34) vanishes by letting / oo. From the above two observations and the fact that wt is even, we have 



lim max > 



k=l 



1 



qk 



[q-l]l 



\q — 1]'' = lim max\^ 
fe=i 

= 0. 



1 



qk 



[q-l]l 



[q-lf 



From (31) and Lemma 13, we have 



lim aA{Ti) = lim maxY^ 



fe=0 



1 - 



qk 



{q-\\l 



{q-\f 



1 + lim max > 

fe=l 



qk 



{q-\\y 



[9-1]' 



= 1. 



Lemma 19: 



lim f3A{n) = 0. 



Proof: Let Cw = {x: w{x) = w}. Then we have 



In the following, we first show that 



< 



exp (nh + wlogg(g' - 1)^ . 



-|:,^E»p(-i^i)G)i.-i'-. 



We have 



E \Cu 
qi 



1 + - 1] exp 



qwT 



[q-l]l 

l + [g-l]exp(-,p^)- 



+ [g-l]exp(-,|g^) 



The first term on the right hand side of (42) is evaluated by 



l + fe-l]exp(-[P^ 



< 



1 + 



10=1 



qwr 



2[q-l]l 



(39) 
(40) 

(41) 



(42) 
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/ WT 

W=l 

< nqexp (-^J 

W — 1 

nql 
<^exp 

q (, P r 
= ^exp(log,--- 



< 



The first inequality comes from the fact that exp(— a;) < 1 — x/2 for < a; < 1/2. The first equality comes 
from (39). The second inequality comes from the fact that [1 + < exp(Za;). The third inequality comes 
from the fact that ri^q^ exp (— is a non-increasing function of w. The fifth inequahty comes from (35). 
From (43), the first term on the right hand side of (42) vanishes by letting I oo. The second term of (42) is 
evaluated by 



l + [g-l]exp(-;|^) 



< E i^^^i^^p 



+ [g-l]exp (-21^) 



where the second inequahty comes from the fact that 

l + [g-l]exp(-5j^) 



^^+?loge(9-l)- 3 



< e-3 



(44) 



and the third inequality comes from (40). From (36) and (44), the second term on the right hand side of (42) 
vanishes by letting I oo. From the above two observations, we have (41). 

Next, we show the lemma by assuming that q = 2. From (32), the fact that wt is even, and Lemma 16, we 
have 



/?aW = E§^E 

w=l fc=0 



1 - 



I 



^ o l*^"-! ( '^kwT\ 



2' 

w=l fe=0 



I J \k 



10=1 fe=0 ^ ' ^ 



From (41), we have the lerrnna for q = 2 

Finally, we show the lemma by assuming that q > 2. From (32), and Lemmas 12 and 17, we have 



/^-w=e¥e 



w=l 



fe=0 



1 



qk 



March 19, 2009 



DRAFT 



< 



w=l ^ fe=0 V LI J / \ / 1 j.^Q \ / 



E\Cw\ \ ^ 



qkwT 

[q - l]l J \k 



(45) 

tU = l fc=0 \ L-i J~/ X'"/ 10=1 

From (41), the first term on the right hand side of (45) vanishes by letting I — > oo. From (40), the second term 
on the right hand side of (45) is evaluated by 

^ \C^\[q - < exp (nh (^) + w[l - t] log,(g - 1)) 



W=l 



< ^lexp [ n^umx^^ [h{0) +n[l - r] logg(g - 1)6'] 



w=l 



< exp l^nh j + [1 - r] log,(g - 1) 

< cxp (1 + log, n + [1 - t] log,(g - 1) + log, ^l,) 
where the third inequality comes from Lemma 14 and the fact that 

[l-r]log,(g-l) < -log,(n-l) 

for all sufficiently large n and q > 2. The fourth inequality comes from Lemma 15. From (35), we have 

1 + logg n + [1 - r] logg(g - 1) + log, S,l -oo 

by letting n — > oo. Then the third term on the right hand side of (45) vanishes by letting n — > oo. Hence we 
have the lemma for g > 2. ■ 



D. Proof of Theorem 3 
We define the set T as 



- -log i,x\Y{x\y)<H{X\Y) + 'f 

(x,y): --\ogiiY\x{y\x)<H{Y\X) + j 
n 

1 



- log nxY{x,y) < H(XY)+-f 
n 

It should be noted that the above definition can be replaced by that defined in [33]. This imphes that the theorem 
is valid for general correlated sources. 

Let {x, y) be the output of correlated sources. We define 

• {x,y)iT (SWl) 

• 3x' ^xsX. x' e CA{Ax),nx,Y{x\y) > nx,Y{x,y) (SW2) 

• 3y' T^y s.t. y' e CB{By),^ix,Y{x,y') > i^x.Y{x,y) (SW3) 

• 3{x',y') {x,y) s.t. x' e Ca{Ax), y' e CsiBy), iix,y{x' ,y') > tix,Y{x,y). (SW4) 

Since a decoding error occurs when at least one of the conditions (SW1)-(SW4) is satisfied, the error probabiUty 
is upper bounded by 



EiTOTxYiA, B) < nxY^Si) + I^XYiSt n £2) + HxYiSl n £3) + I^XY{St n £4), 



(46) 
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where we define 

Si = {{x,y):(SWi)}. 
First, we evaluate Eab [Mxr(^i)]- From Lemma 26, we have 

Eab[^ixy{£i)]<^ (47) 

for all sufficiently large n. 

Next, we evaluate Eab [i^xy{£i H £2)] and Eab H £3)]. Since 

^ix\Y{x'\y) > fix\Y{x\y) > 2-"[^(^l^)+^l 

When (SWl) and (SW2), we have 

[giy)\{x}]nCA{Ax)y^(l}, 

where 

e(y)^{x:/.x|^(x|y)>2-"[W)+7]}. 

From Lemma 1, we have 

Eab if^xYiS^nS^)] = ^ fixYix,y)pAB {{{A.B) : (SW2)}) 

ix,y)eT 

< Yl ^^xY{x,y)pA{{A■.[giy)\{x}]nCA{Ax)^<D}) 

{x,y)eT 



< 

(a!,i/)eT 



2n[//(X|y)+7]^^ 

M 



< X] I^XY{x,y) 

llm^l 

S 

< 4 (48) 

for all sufficiently large n by taking an appropriate 7 > 0, where the last inequality comes from (14) and an 
(aA,/3A)"hash property of {A,pa)- Similarly, we have 

Eab [mxy{£! n £s)] < 2"[^(^l^)+7l|3;|-'« + 

< I (49) 

for all sufficiently large n by taking an appropriate 7 > 0, where the last inequality comes from (15) and an 
(as,/3s)-hash property of {B,pb). 
Next, we evaluate Eab n £4)]. When (SWl) and (SW4), we have 

[g \ {ix,y)}] n [CAiAx] X CB{By)] ^ 0, 

where 

g = \(x,y): fXxYix,y) > 2-"[^(^m7l} . 
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Applying Lemma 1 to the joint ensemble {A x B,pab) of a set of functions AB : x X'-^ x y'-'^, 

we have 

Eab [i^xy{£! n £4)] = J2 y^PAB {{{A B) : (SW4)}) 

< ^^XY{x,y)pAB {{{A,B) : [Q \ {{x,y)}] n [Ca{Ax) x CsiBy)] ^ 0}) 

{x,y)eT 

\G\aAB 



< l^xY{x,y) 



|Im^||ImB| 



'AB 



< ^ (50) 

for all sufficiently large n by taking an appropriate 7 > 0, where the last inequality comes from (16) and an 
(Q:AB,/3^B)-hash property of {A x B,pA'X- Pb)- 
Finally, from (46)-(50), for all 5 > and for all sufficiently large n there are A and B such that 

Errorxr(^,-B) < S. 



E. Proof of Theorem 5 

For (3a satisfying lim„^oo (^A{n) = 0, let k = {^(n)}^! be a sequence satisfying 

lim K,{n) = 00 

lim K{n)pA{n) = 



log K;(n) 
lim ^ ^ ' = 0. 



n— >oo Tl 



For example, there is such k by letting 

K{n) = < 



if pA{n) = o{n-^) 
otherwise 



L V/3a(«)' 

for every n. If /3a (j^) is not o there is k' > such that PA{n)n^ > k' and 

logK(n) logft^ 



< 



2n 
2n 

^ log n — log k' 
~ 2n 

for all sufficiently large n. This implies that k satisfies (53). In the following, k denotes K{n). 
Let e = SB — £a- Then, from (53), there are 7 and 7' such that 



(51) 
(52) 

(53) 



< 7 < ^/2^log\Z\ < £ 



(54) 
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< y < 2e (55) 

Vwizh'h) < ^ - ^ (56) 

Vx\zwh'h + 2e)<e^-^-^. (57) 

for all sufficiently large n. It should be noted here that there is such 7 and 7' by assuming (18). 

Let z be the output of channel side information, and x and y be an input and an output of the channel, 
respectively, and m be a message. From (55), we have Tw^z,j'{z) 5^ for all z and sufficiently large n. Then 
we have 

\%V\Z,2eiz)\ > \Tw\Z,-i'{z)\ 

> 2n[H{W\Z)-r,y,\z{'i'\y)] 

> y/^\lmA\\lmB\ 

for all z G Tz,7 and sufficiently large n, where the first inequality comes from (55), the second inequality 
comes from Lemma 27, the third inequality comes from (56), and the fourth inequality comes from the fact 
that Im^ c W'-^ and ImB C W''^. This implies that for all z € there is Tw\z{z) C Tw\z,2e{z) such 
that 

for all 2: e 7z,7 and sufficiently large n. We assume that T^^zi^) satisfies the assumption described in 

Lemma 5. Similarly, from (57), we obtain Tx\zw{z,w) C Tx\zw,2e^{z,w) such that 

^ . \Tx\zw{z,w)\ 

< — < 2Jk (59) 

\lmA\ 

for all {z,w) e Tzw,2e and sufficiently large n. 
We define 

• z G Tz,,, (GPl) 

• gAB{c, m\z) G rw\z{z) (GP2) 

• 9a^\z, 9ab{c, m\z)) G Tx\zw{z, 9ab{c, m\z)) (GP3) 
•ye 7^1x2,7(51(^12;, 5AB(c,m|z)),z) (GP4) 
•ffA(c|y) =ffAB(c,m|z). (GP5) 

Under condition (GP5), we have 

V~^(y) = BgA{c\y) = BgAB{c,m\z) = m, 
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which implies that the decoding succeeds. Then the error probabihty is upper bounded by 
F,irorY\xz{AB,A, c,c) 

= X! PM(m)/iz(z)/ii'|xz(y|ffl(c|z, 5ab(c, m|z)), z)x{gA{c\v) + QAsic, m|z)) 

Tn,y 

< PMYz{S^) + PMYz{Sl n S^) + PMYz{Sl n ^2 n SI) + PMYZ{SD + PMYz{Si H ^2 H ^3 H ^4 H S^), 

(60) 

where 

<Si = {(m,y,z) :(GP*)}. 

Let (5 be an arbitrary positive number. 

First we evaluate -B^sacc \Pmyz{Si)\ and E^^^^q IpMYzi^i)]. From Lemma 26, we have 

5 



EABACd \PMYz{St)] < ^ (61) 



E,,sAc8\PMYz{Sd]<^ (62) 



for sufficiently large n. 

Next we evaluate -BAscbMyz(«5i fl S2)] and -B^g[pMyz(«5i fl <S2 fl (Sf)]. From Lenmia 5, we have 
EABc\PMYz{Sir\S2)]= ^ /iz(z)pABCM ({(^,-B,c,m) : s'AB(c,m|z) ^ Tvi/|z(z)}) 



< 



|Im^||Img|[/3AB + l] 2-"^|W|'-^+''^ 
+ ^ IWIIImSI 



/3ab + 1 2-"^|W|'^+''^ 
_aAB 1+ ^ + |W||Ime| 

< ^ (63) 


for all sufficiently large n, where the second inequaUty comes from (58), and the last inequality comes from 
(51) and the properties of [olabi^ab) |W|'^/|Im^|. Similarly, by using (59), we have 

E^Q \PMYz{s, ns^n 53^)] <a^-l + + ' I 

V |Im^| 

< \ (64) 


for all sufficiently large n. 

Next, we evaluate E j^^^.cd^^'^ ^^^^ n ^2 n ^3 fl ^4 fl Sff\. In the following, we assume that 

• 2; e Tza 

• w G Tw\z{z) c Tw\zae{.z) 

• X GTx\zw{z,w) C Tx\zW,2ej;{z,w) 

•ye Tyixz^jix, z) = Tyixzwni^^ 
•9A{c\y) ^ w, 
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where the relation TY\xz,-y{x^ = '^Y\xzw,'y{Xi comes from (17). From Lemma 23, (18), and (54), we 
have (a;, y, z, w) G Txvxzwfie^- Then there is w' £ Ca{c) such that w' ^ w and 

IJ'W\y{w'\v) > fJ-wiviwly) 
^ I^WY{w,y) 
t^Y{y) 

2~n[H(W,Y)+Cywi6e^)] 
> 2-nlH(W\Y)+2<:ywi6sx)] ^ 

where the second inequaUty comes from Lemma 26. This impUes that 

[g{y)\{w}]nCAic)^9, 

where 

g{y) ^ {w' : i,w\Yiw'\y) > 2-«W^I^)+2C:.w(6e,-)]| _ 

Then, we have 

^ABAcc \PMYziSi n 52 n ^3 n 54 n SI)] 

^^ABACMC H ^^(^) X{9AB{c,m\z) = w) 



X] Mr|xz(y|a;, z)x{gA{c\y) ^ w) 



— ^ABACMC 



X] X(5'l(c|z,iu) =a;) ^ ^ 

a!e7'x|zvv(2,-«') l/e7Vixz,7(a!,«) 

X X ^('^■^ = c)x{Bw = m) 



^x = c) X /^v|xz(y|a:,2;)x(5A(c|y) 9^ tu) 



1/ 



= X 13 ^ 



zeTz,-, 

■Eac 



E 



I^Ty\xz,-,{3:,z) 
I v -. (a;,z) 



■(y|a;,2;) 



llmBIIIm^l ^ ^ ^ ' ^ ^ r-i-l^z, 

\Q{y)\{w}\^CA{c) + %\\ 
■ PAC <{Ac): > 

^ we Ca{c) J y 



x(ffA(c|y) 7^ '!i;)x(Aw^ = c)i;s^MC [xiBw = m)xiAx 

^ iy\x,z) 



llmSIIIm^l ^ ^ ^ ' ^ ^ ^ ' 

I II I zeTz,^ w^T„\z{z)xeTx\zw(z,w)y^TY\xz,i(x,z) 

-2n[//(W|y)+2Cyw(6e^)]Q^ " 

_ |Im^|2 llmy^l 



|xz(y|a;,z) 



< 



|Im^|2 \\^A\_ 

- 2n[H(W|r)+2Cyw(6e^-)]c,^ 1 ^ / ^ 1 ^ 1 

^ ^fiA^ 2. iiz{z) lini^llimBI ^ ^ 
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ll^l ^'^^^ 

< I. (65) 

where the third inequahty comes from (9), the fourth inequaUty comes from Lemma 3 and the fact that 

\G{y)\ < 2"l^(^l'^)+2^yw(6<r^)]^ 

the sixth inequality comes from (58) and (59), and the last inequaUty comes from (19), (52), and the properties 
of (aA,/3A) and |W|'-^/|Im^|. 

Finally, from (60)-(65), we have the fact that for all 5 > and sufficiently large n there are A G A, B G B, 
A G A, cG ImA, and c such that 

ETiorY\xz{AB,A,c,c) < S. 



F. Proof of Theorem 7 
We define 



and assume that 7 > satisfies 



• {x, z) G Txz,i 
•9a{c\x) g 7V|xz,2£^(a;,z) 

• 9AB{c,BgA{c\x)\z) =gA{c\x) 

^+^\og\X\\Z\ <eA- 



(WZl) 
(WZ2) 
(WZ3) 

(66) 



We prove the following lemma. 

Lemma 20: For any {x,z) satisfying (WZl) 

PABC {{{A, B, c) : (WZ2), not (WZ3)}) < 



■ les-eA-'^CyziSEA)] |3;|'.A+' 



'as 



+ Pb. 



\lmA\\ImB\ 

Proof: If (x, z, A, B, c) satisfies (WZl) and (WZ2) but not (WZ3), there is y' G Cab{c, BgA{c\x)) such 
that y' 7^ gA{c\x). Then, from (66) and Lenmias 23 and 25, we have {x,gA{c\x),z) G Tyxz,3sa and 

^J'Y\z{y'\z) > HY\zi9Aic\x)\z) 

^ IJ,yz{9a{c\x),z) 
fJ-ziz) 

2-n[H(Y,Z)+Cyz{3eA)] 

> 



2-n[H(Z)-Cz(3eA)] 
^ 2-nlH(Y\Z)+2Cyz(36A)] _ 



This impUes that 



where 



[G \ {gA{c\x)}] n Ca{c, BgA{c\x)) ^ 0, 
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Let c = 9a{c\x). From Lemma 4, we have 

PABC {{{A, B, c) : (WZ2), not (WZ3)}) < pabc {{{A, B, c):[g\ {y^, J] n Cab{c, By^J ^ 0}) 

\g\aB 

- |Im^||ImS| 

2n[HiY\Z)+2Cyz{3eA)]aj^ 

- Ilm^lllm^l 

2-n[£B-£^-2Cy2(3£^)]|-y|U+'BQ,^ 

" \ImA\\ImB\ ^ 

where the second inequahty comes from Lenmia 4 and the third inequality comes from the fact that 

\g\ < 2"['F^Ci^l^)+2Cy2(3£^)], 



Proof of Theorem 7: Let ¥iVvorxz{A, B,c) be defined as 

Errorxz(A,-B,c) = Hxz {{{x,z) : {x,ip~'^{ip{x),z),z) ^ TxYZ,3e^}) ■ 

Since 

{x,ip~^{ip{x),z),z) = {x,gAB{c,BgA{c\x)),z) 
= {x,gA{c\x),z) 

€ TxYZ,3eA- 

under conditions (WZ1)-(WZ3), then we have 

Errorxz(A, B, c) < fixziS^) + l^xz{Si n S^) + nxz{Si n ^2 n ^3^), (67) 

where 

Si = {{x,z):(WZi)}. 

Let ^ > be an arbitrary positive number. 

First, we evaluate Eabc [p^xziSf)]. From Lemma 26, we have 

Eabc WziSl)] < - (68) 

for all sufficiently large n. 

Next, we evaluate Eabc fl «S2)]. From (20), we have 

IJ'XYzix, y, z) = iJ.xz{x, z)iJ.Y\x{y\x) 

_ IJ'Xz{x,z)iJ.xY{x,y) 
fJ-xix) 

= iJ.xY{x,y)iJ,z\x{z\x) 

and 

arg max nxY{x,y') = aig max ij.xY{x,y')nz\x{z\x) 
y'eCB{c) y'eCB{c) 
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= arg max nxYzix,y' , z). 
y'eCBic) 

This implies that ML coding by using fxxY is equivalent that using fixYZ- Since 7 > satisfies (66), we have 
the fact that there is 7' > such that 

'ny\xz{i\l) <£A-1 

for all sufficiently large n. We have Ty\xz,'^'{x^ -2) 7^ for all (a;, z) e Txz,-i and sufficiently large n. Then, 
from Lemma 27, we have 

\TY\XZ,2eAix,z)\ > \Ty\XZ,y{x,z)\ 

> 2n[H{Y\XZ)-r,y^;^z('^'\'l)] 

> 2n[H(Y\XZ)-eA+-i\ 

> 2"^|Im^| 

for all (x, z) e Txz,-^ and sufficiently large n. This implies that there is T(x, z) C 'T^Y\xz,ieA ^"''^ ^^'^ 

|r(a;,2)| > 2"T|Im^|. (69) 
We assume that T(x, z) satisfies the assumption described in Lenmia 5. Then from Lemma 5, we have 
EABc\ii'Xz{Sxr\Sl)\= ^ /ixz(a;,z)pAC ({(Ac) : 5(a(c|x) ^ Ty.|xz,2e^(a;,z)}) 

(a!,z)eTxz,T 



< XI lJ-xz{x,z) 

< X] Mxz(a:,z) 



l^(^^=,^)l iwi 



aA - 1 + 2-"T [/?^ + 1] + 



2-"'^-4|3;|'^ 



|Imyl| 



< - 1 + 2-"T [/3a + 1] + 



5 

< - 
- 3 



\lmA\ 



(70) 



for aU sufficiently large n, where the third inequaUty comes form (69), and the last inequality comes from the 
properties of {cxajISa} and IJ^I'-^/IIm^l. 

Finally, we evaluate Eabc [mxz(<5i fl <S2 fl S^)]. From Lemma 20, we have 

Eabc [l^xz{Si n 52 n ^3^)] < ^Wbc {{{A, B, c) : (WZ2), not (WZ3)}) 

{x,z)erxz,y 



< 



< 



|Im^||IinB| 



(71) 



for sufficiently large n, where the last inequaUty comes from (21) and the properties of (as, /S^), |3^|'-^/|Im>l| 
and \y\^^/\lmB\. 
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From (67)-(71), we have the fact that for any 6 > and for all sufficiently large n there are A, B, and c 
such that 

ErrorxY(^, -B,c) < S. 



From Lemma 24, we have 

Pn{x,fn{y,z)) 



n 



^ iy^yz{x,y,z)p{x,f{y,z)) 

{x,y,z)eXxyxZ 

^ X] [lJ'XYz{x,y,z) + V6e^p{x,f{y,z)) 

{x,y,z)eXxyxZ 

< |J'XYz{x,y,z)p{x,f{y,z)) + \X\\y\\Z\p^^^^/6e^ 

(x,y,z)eXxyxZ 

= ExYz[p{X,f{Y,Z))] + \X\\y\\Z\ 

Pmax 



for (x, y, z) e 7x-rz,3£^- Then we have 

Exz [Pn(^", /n(y-Hy'(^"), Z^), Z"))] 



< i;xFz [p(x,/(F,z))] + \x\\y\\z\p^^^^j%i^ + 5p^ 

< ExYz [p{X, f{Y, Z))] + 3\X\\y\\Z\ 



for all sufficiently large n by letting 



s<[3-VQ]\x\\y\\z\^. 



G. Proof of Theorem 8 
We define 



and assume that 7 > satisfies 



•9Aic\y) e TzlXY,2eA^^y) 

• QAsic, BgAic\y)) = gA{c\y) 

• gg{Bx\gAB{c,BgA{c\y))) = x 



7+ V27log|A'||3^| <£^. 



We prove the following lemma. 

Lemma 21: For any {x,y) satisfying (OHOl), 

Pabbc {{{AB,B,c) : (OH02),(OH03), not (0H04)}) 
Proof: We define 



(OHOl) 
(0H02) 
(0H03) 
(0H04) 

(72) 



< 



2-ra[eB-2C^z(3e^)]|^|(gQ,^ 



llmSI 



+ /3s. 



^A,B,B,c = 9B^Bx\gAB{c, BgA{c, x))) 
ZA,B,c = 9ab{c, BgA{c,x)). 
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Assume that conditions (0H01)-(0H03) are satisfied but (0H04) is not. From Lemma 23 and (72), we have 
{x,y,gA{c,y)) e TxYZ^Ze^ and there is x' G Cg{Bx) such that x' ^ gg{Bx, gAsic, BgA{c\x))). From 
Lemma 26, we have 

fiX\z{x'\ZA,B,c) > I^X\z{Xa^b,B,c\^A,B,c) 
^ fJ'Xz{x^ j^ g ^,ZA,B,c) 

fJ-ziz) 

2-n[HiXZ)+Uz{3eA)] 

- 2-"[^f(^)-f2(3^^)l 

^ 2-nlH{X\Z)+2Cxz{3eA)] _ 

This impUes that 

'Q{za,b,o) \ {xa,b,b,o}] nCg{Bx) ^ 0, 
g{z) ^ {x' : iJ.x\z{x'\z) > 2-"[^^(^l^)+2f--(3e^)l} . 

From Lemma 1, we have 

Pabbc {{{A,B,B,c) : (OH02),(OH03), not (0H04)}) 

— Pabbc 

{{iAB,B,c) : [e(^A,B,c)\ W,s,g,c}] nC5(Bx^^3_5 J ^0}) 
= Yl Pa>bA^, B, c)pg ({S : [g{zA,B,o) \ {Xa,b,bJ] ^ Cg{Bx^ j, g J ^ 0}) 



where 



A,B,c 



< X] PA,B,c{A,B,C 

A,B,c 

2nlH{X\Z)+2(:xz{3eA)]a 



Q{ZA,B,c) \ {Xau^bJ 



Oif 



llmSI 



< 



|ImS| 

2-Tl[£g-2fAf2(3£.A)]|;^'pgQ;, 



where the last inequality comes from the fact that 

\G{zA,B,o)\ < 2"[^(^I^)+2C^^(3-^)1, 

for all A, B and c. ■ 

Proof of Theorem 8: Under the conditions (0H01)-(0H04), we have 

if- ^{ipx{x),ipY{y)) = gg {Bx , gAB (c, BgA (c| y) ) ) 

= X. 

Then the decoding error probability is upper bounded by 

Errorxr(^, B, B, c) < iixy{S1) + Hxy{S^) + I^xy{Si n ^2 n S^) + i^xy{Si n ^2 n ^3 n (73) 
where we define 

Si = {{v,x):iOUOi)}. 
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From (22), we have 



IJ,xYzix,y,z) = iixY{x,y)iJLz\Y{z\y) 
^ fJ'XY{x,y)iiYz{y,z) 
^J'Yiy) 

= iJ'X\Y{x\y)iJ-Yz{y,z) 



and 



arg max ij,z\y (z' \y) = a,rg max iiYz{y,z') 
z'eCA(c) z'eCA{c) 

= arg max /xrz(y, z')iJ,x\Y{x\y) 
z'eCAic) 

= arg max ij,xYz{x,y, z') 

= arg max ij,z\xY{z'\x,y) 
Z'eCAic) 

This impUes that ML coding by using Hz\y is equivalent to that using fiz\XY- By applying a similar argument 
to that in the proof of Theorem 7, we have 

EABBci^^xYiSt)]<l (74) 

^^ABSC < - 1 + 2-"^ [/?^ + 1] + 



< ^ (75) 



E^ssc n 5. n ^3^)] < ^^^^^^ ^ + 

< ^ (76) 
for all sufficiently large n by assuming (23) and (72). Furthermore, from Lemma 21, we have 

^ABBc [f^xYiSi n 52 n 53 n S^)] 

< t^XYix, y)PABBC ({(^. c) : (OH02),(OH03), not (0H04)}) 

{x,y)eTxY,y 



2-Ti[£g-2CArz(3e^)]|_;j^|igQ,^ 



llmBI 



< f^xYix,y) 

{x,y)eTxY,~, 
2-n[s§-2CxziSeA)]\;^\lBa- 

|ImB| ^ 

< ^ (77) 

for all sufficiently large n, where the last inequaUty comes from (24) and the properties of {ag,f3g) and 
\X\^s/\lmB\. 

From (73)-(77), we have the fact that for all 5 > and sufficiently large n there are A, B, B, and c such 
that 

ErrorxrlA-B,^,^) < S. 
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VII. Conclusion 

In this paper we introduced the notion of the hash property of an ensemble of functions and proved that an 
ensemble of g-ary sparse matrices satisfies the hash property. Based on this property, we proved the achievability 
of the coding theorems for the Slepian-Wolf problem, the Gel'fand-Pinsker problem, the Wyner-Ziv problem, 
and the One-helps-one problem. This impUes that the rate of codes using sparse matrices combined with ML 
coding can achieve the optimal rate. We believe that the hash property is essential for coding problems and 
our theory can also be applied to other ensembles of functions suitable for efficient coding algorithms. In other 
words, it is enough to prove the hash property of a new ensemble to obtain several coding theorems. It is 
a future challenge to derive the performance of codes when ML coding is replaced by one of these efficient 
algorithms given in [1][18][1 1]. It is also a future challenge to apply the hash property to other coding problems. 
For example, there are studies of the fixed-rate universal source coding and the fixed-rate universal channel 
coding [31], and the wiretap channel coding and the secret key agreement [32]. 

Appendix 

Method of Types 

We use the following lemmas for a set of typical sequences. It should be noted that our definition of a set 
of typical sequences is introduced in [15][41] and differs from that defined in [6] [4] [16] [48]. 
Lemma 22 ([6, Lemma 2.6]): 

- log — T-r = H{uu) + D{uu\\iJLu) 
n nu{u) 

n ^j.u\v{u\v) 

Lemma 23 ([41, Theorem 2.5]): liv£ Ty^-y and u G Tu^y^y^v), then {u, v) £ Tuv,-y+'y'- If {u, v) € Tuv,-y, 
then u € ^,7- 

Proof: The first statement can be proved from the fact that 

D{Puv II fiuv) = D{y^ II nv) + D{vu\^ \\ nij\v\i^^)- C^^) 
The second statement can be proved from the fact that 

D{vv II IJ-v) < D{vu,v II mv), 

which is derived from (78) and the non-negativity of the divergence. ■ 
Lemma 24 ([41, Theorem 2.6]): If u E Tu^^, then 

l^^u(w) — Mi/('")l ^ '\/27, for all u gU, 

Vuiu) = 0, if nu{u) = 0. 
Proof: The lemma can be proved directly from the fact that 



ueu V °^ 

where e is the base of the natural logarithm (see [4, Lemma 12.6.1]). 
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Lemma 25 ([41, Theorem 2.7]): Let < 7 < 1/8. Then, 



1, 1 

- log2 



H{U) 



< Cw(7) 



for all ti e Tf/^, and 



n nu\v{u\v) 



H{U\V) 



(79) 



(80) 



for V e 7V,7 and u G T^^y^y{v), where CwCl) and Cuivil'll) defined in (2) and (3), respectively. 
Proof: From Lemma 22, we have 



- log2 - H{V) 



<D{v^ \\lJiv) + \H{v^)-H{V)\. 



We have (79) from [6, Lemma 2.7]. 
From Lemmas 22 and 24, we have 

1 



- log2 ;— 

n iiu\v(u\v) 



H{U\V) 



< £>(z/„|„ II iJ.u\vWv) + \H{v^\^\v^) - H{iiu\v\i'v)\ + \H{nu\v\vv) - H{U\V)\, 



and 



\H{iiu\vWv) - H{U\V)\ < v'27log2 l^^l, 



respectively. We have (80) from the above inequaUties and [6, Lemma 2.7]. 
Lemma 26 ([41, Theorem 2.8]): For any 7 > 0,and v G V", 

fiu\vi[ruivA'")r\v)<2--^''-^"-K 

where Xu and Xuv are defined in (1). 

Proof: The lemma can be proved from [6, Lemma 2.2] and [6, Lemma 2.6]. 
Lemma 27 ([41, Theorem 2.9]): For any 7 > 0, 7' > 0, and v G Tv,j, 



-\og2\Tu,^\-H{U) 



-log^\Tuiv,Yiv)\-H{U\V) 



< Vuil) 



where 77^(7) and ??i^|v(7'|7) are defined in (4) and (5), respectively. 

Proof: The lenmia can be proved in the same way as the proof of [6, Lemma 2.13]. 
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